Performance Metrics and Their Extraction Methods for Audio Rendered Mathematics

We introduce and compare three approaches to calculate structureand content-based performance metrics for user-based evaluation of math audio rendering systems: Syntax Tree alignment, Baseline Structure Tree alignment, and MathML Tree Edit Distance. While the first two require “manual” tree transformation and alignment of the mathematical expressions, the third obtains the metrics without human intervention using the minimum edit distance algorithm on the corresponding MathML representations. Our metrics are demonstrated in a pilot user study evaluating the Greek audio rendering rules of MathPlayer with 7 participants and 39 stimuli. We observed that the obtained results for the metrics are significantly correlated between all three approaches.